Testing the Effectiveness of the Diagnostic Probing Paradigm on Italian Treebanks

The outstanding performance recently reached by neural language models (NLMs) across many natural processing (NLP) tasks has steered the debate towards understanding whether NLMs implicitly learn linguistic competence. Probes, i.e., supervised trained using NLM representations to predict properties, are frequently adopted investigate this issue. However, it is still questioned if probing classification really enable such investigation or they simply hint at surface patterns in data. This work contributes presenting an approach assessing effectiveness of a suite aimed testing knowledge encoded one most prominent NLMs, BERT. To aim, we compared probes when predicting gold and automatically altered values set features. Our experiments were performed on Italian evaluated BERT’s layers for sentences with different lengths. As general result, observed higher prediction values, thus suggesting that model sensitive distortion feature values. our also showed length sentence highly influential factor able confound model’s predictions.